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ABSTRACT 

The RMC Model A tncrm- referenced) for evaluation of 
Title I programs is based upon the equipercentile assumption — that 
students maintain their percentile rank ever a one-year period, 
provided that no special instrucional intervention is introduced. The 
control group, essentially the sample used to standardize the 
achievement test, represents the no-treatment expectation. The 
treatment effect is defined as the average posttest performance minus 
the average pietest performance. Title I participants are usually 
selected based upon low pretest scores, and may tias the results cf 
Model A evaluation because lew ability students generally regress 
toward the mean without any intervention. Two models which cculd 
estimate the no-treatment expectation and which might solve thes^ 
problems were investigated: H) Hurray's recursive path analytic 
paradigm, encompassing three achievement measures, a group factor, an 
individual differences within group factor, and an prror component; 
an-l (2) an extension of the one independent variable regression 
situation. Results indicated that fcr local districts with fewer than 
500 students per grade, the no-treatment expectation would prcbably 
be overestimated. These two methods would be accectable for use in 
districts with 2,000 or more students per grade. The use of local 
correlations, b^^-^^d on ncr-Title T participants, was also suggested. 
(GDC) 
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As a result of the 1974 Congressional amendment of the law governing 
ESEA Title I programs, RCM Research Corporation developed a set of alterna- 
tive evaluation models intended to provide a standardized frameii/ork from 
which to judge the effects of the Title I effort at the Tederal level. 
Each of the models was presented to the United States Office of Education 
(USOE) as a means of obtaining an unbiased estimate of the no-treatment 
expectation, provided that the assumptions of each model could be adequately 
satisfied. The three mdels developed by KMC Research Corporation are the 
Control Croup Model (Model B) , the Special Regression Model (Model C) and 
the Norm Referenced Model (Model A). 

The intent for all three models is to provide an unbiased estimate of 

the Title T treatment effect which RMC has defined as 

Obtained Expected 
Treatment effect = Posttest performance - Posttest performance. 

The models differ primarily in the way in which the ^'expected posttest 
performance" is obtained. RMC defines this '^expected posttest performance*' 
or "no-treatment expectation'* as the performance of Title I students who 
had not received any special instructional Intervention. 

In this paper we focus on liodel A, the Norm Referenced Model because 
it is the most popular model among LEAs at the present time, and because it 
appears to be the model most prone to biased estimates of the no-treatment 
expectation. The major assumption underlying Model A is the "eqiiiporcent i I e" 
assumption. The basic concept is that students will maintain their per- 
centile rank over a one vear instructional d provided that no special 
instructional intervention is introduced. 

The control ^roiip of Model A is essentially the norm group used for a 
particular standardized achievement test, since the national percentiles are 
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obtained from this source. Furthermore, since the pretest percentiles are, 
by definition, the no-treatment expectation under tlic equipercent i 1g assump- 
tion, the Title I treatment effect for Model A is simply defined as the 
average posttest performance minus the average pretest performance. 

One of the major disadvantages of Model A is that Title I participants 
are generally selected on the basis of poor academic performance. And v/e 
know that low scoring students are likely to regress towards the population 
mean on a second measurement even with no special instructional interven- 
tion. In an effort to avoid this problem, Tallmadge (1976) recommended 
that selection of Title I participants be based upon some measure other 
than the pretest. The assumption that Tallmadge (1976) made is that all of 
the regression that is going to occur would do so between the selection 
measure and the pretest. Therefore, an unbiased estimate of the no-treatment 
expectation could be obtained. 

Glass (1978) and Murray (1978) clearly demonstrate, however, that the 
use of a separate selection measure does not guarantee the elimination of 
all regression between the pretest and the posttest, although they do note 
that a reduction in the magnitude of the regression is to be expected. 
Murray (1978) further indicates that the only time that all regression 
would be eliminated when a selection test is used, is when the selection- 
pretest correlation qunl to the selection-posttest correlation. 

The problem then, is to develop an unbiased estimate of the no-treatment 
expectation given thnt the selection procedure into Title I is fiillv under- 
stood. Murray (19 78) has proposed one such es Lima Lien procedure which 
may lead to less biased estimates. In addition, tlicre is tlie tr^iditionnl 
multiple linear regression approach to this estimation task assuming some 
degree of multivariate normality and linearity. 

The purpose of this paper i.s to investigate, through n limited 
simulation, the sampling distributions of these two estimation models and 
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to determine the magnitude and direction of bias, if any, as well as the 
size of the standard errors under the assumptions of multivariate normality 
and linearity. The concern of greatest importance is that an estimation 
model will lead to large standard errors and consequently lead to frequent 
underestimates of the actual treatment effect. 

The following notation will be used throughout this paper: 
= Selection test given at time i. 

Xj^ = Pretest given at time 1. 

X2 = Posttest given at time 2. 

r = Will represent population correlations. 

Z = Unit normal standard scores . 

Z = Estimated mean. 

Z = Obtained mean. 

Estimation Procedure s 

The approach adopted by Murray (1978) for deriving the no-tren tmen t 

expectation under a selection, pretest-posttest design, was to use a 

set of structural equations in a recursive path analytic paradigm. In 

addition to the three achievement measures, tlie paradigm included a group 

factor, an individual differences within ^roup factor and an error 

component. The result of this path analytic approach indicates that an 

unbiased estimate of the no-treatment expectation can be obtained using 

the ratio of two correlations; the correlation of selection with posttest 

(r^ ^ ) over the correlation of selection with pretest (r^ x ^ °^ ^x x ^^x x ' 

0 2 0 1 0 2 0 1 

times the estimated pretest mean. Murray poin'Ts out that this estimate takes 

Into account the distance of the treatmenc group mean from the population 

mean as well as the temporal erosion expressed by the ratio. Using standard 

score form, Murray's (1978) estimate of the no-treatment expectation is 
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(1) Z, = (r /r ) 7., 

where Z2 is the estimated posttest mean, 

Zj^ is the estimated pretest mean, 

and Z, is obtained using Z, = r Z_ . 

It becomes apparent when examining (1) that the regression effect from 

pretest to posttest will equal zero onlv when = r • 

0 1 0 2 

As with all estimates of regression, this model is parametc dependent 
in that population correlations, means and variances (or very good estimates 
of them) are required. In a sense Murray's procedure is a stagewise 
approach in that Z, must be estimated from Zq prior to estimating Z2. That 
is, we first determine the regression from selection to pretest and then we 
compute the remaining regression from pretest to posttest using the ratio 
in (1). Actually, this stagewise process is not necessary since the same 
result can be obtained directly from 

(2) - r,^,^ J„. 

In fact this linear form appears in Murray's derivation one step prior to 

the ratio form. Through a simple substitution we have 

Zo = (r /r ) Z, 
1 XqX2_ x^x^ 

^ V2' Vl ^^^0^1 
= r "Z- . 
V2 ° 

This linear form, however, is less Informative when a selection test is in 

use. For example, it would not be apparent from (2) that pretest to 

posttest regression is eliminated only when r^^ ^ ~ ^x x * 

0 1 0 2 

The second estimation method examined in this paper is simply n direct 
extension of the one independent variable regression situation. When students 
are selected only on the pretest (with no prior preselection) then pretest 
to posttest regression is estimated directly from 

^2 = 
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When a selection test is added to the picture, the logical extension is a 

multiple regression of the form 

(3) Z2 = bi Zq + b2 \ , 

"^^"■^ ''l = ^"^x X ' ""x X ""x X ^ " "^x X 

V2 ''l''2 Vl O^'l 

and = (r - r r )/ 1 - rj . 

2 x^X2 XqX2 XqX^ XqX^ 

As in the case of (1), this form is parameter dependent in the correlations. 
However, in this case there is no need to estimate Zj^. Both the ratio form 
nnd the multiple linear form require the assumption of linearity and 
homogeneity of regression. 

The questions of interest in this paper are 1) are the standard errors 
of the sampling distributions of (Z2 - Z2) sufficiently small to eliminate 
any major risk of underestimating the treatment effect, and 2) does varying 
the correlations significantly affect these standard errors? 
Simulation Procedure 

In order to simulate the regression effect of interest, it was necessary 
to produce three normally distributed random variables with specified 
intercorrelations . These variables could then serve as selection, pretest 
and posttest scores. The method used in this study is described in detail 
by Kaiser and Dickman (1962). The procedure uses component analysis for 
generating a sample score matrix (Z) and a correlation matrix (R) when 
given a specified population correlation matrix (R) . The fundamental 
postulate of component analysis is 

Z = FX, 

where F is any factoring of R and X is a population score matrix. The 
components in F must be uncorrelated such as principal components or 
square root components. The sample score matrix is then produced using 

Z = FX 
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where X is a matrix containing sample values from a normally distributed 
random variable. The variables in X are uncorrelated but the variables in 
Z are correlated according to the population correlation matrix (R) . 

The computer routine used in this study is GGNRM and it is found in 
the International Mathematics and Statistical Library (IMSL) package. It 
is an accurate, but slow and costly routine. The primary inputs to GGNRM 
are the population correlations, the number of caSes to be generated and a 
seed to start the random number generator. 

In this study, 3x3 correlation matricies were used and a 3 by 500 
score matrix was generated. Descriptive statistics were then accumulated 
including the reproduced (sample) correlation matrix. This procedure was 
executed for 300 iterations in order to obtain a reasonable estimate of 
the sampling distributions of interest. This study is limited to a sample 
size of 500 due to the excessive time required. 

The procedure involves using the first of the three score vectors as 
the selection test and selecting easels on the basis of a cutoff score. In 
this case, we are dealing with a multivariate normal distribution and 
therefore a Z of -.52 was used to select approximately the lower scoring 
30% of the cases. After each iteration, we computed descriptive statistics 
on the total 500 cases as well as on the lower scoring 30% or about 150 
cases that are of special interest. 

Since a multivariate normal generator was used, it was a relatively 
simple matter to derive the expected values of all three variables. The 
first stop involved determining the population mean of the lower scoring 
30 percent. This is accomplished using 

Z = ii;(Z)/<|)(Z), 

where i|^(Z) is the height of the ordinate at a Z of -.52 and *(Z) is the 
probability associated with a Z of -.52. Theroforc we expect that the mean 
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of the selection test variable will be 

Z = ^(-.52) / (H-.52) 
- .3485/. 3015 
^ -1.1559. 

Or, in terms of the NCE metric with mean of 50 and a o of 21.06 we have 

Z = 21.06 (-1.1559) + 50 = 25.6570. 
In order to determine the expected pretest and posttest values, it is 
necessary to specify the correlations. For example in the run that used 
.9, .8, .9 as selection - pre, selection - post, and pre - post correlations 
respectively, the expected pretest value is 

1 XqXj^ 0 
= .9 (-1.1559) 

= -1.04, 

or in NCEs Z = 28.09. The expected posttest value can then be 

determined using either estimation formula (1) or (3) since all input 
values are the same. Using (1) in this example we have 

^2 = ^x X / ^x X 
0 2 0 1 

= .8/. 9 (-1.04) 
= -.92, 

or in NCEs Z2 = 30.53. 

The values of particular interest in this study, however, are the 
differences between obtained and estimated posttest means and the 
corresponding standard errors for these differences. 
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Results and Discussion 

All computations were performed using unit normal standard scores 
(Z scores). For reporting purposes, these Z scores were then transformed 
to the NCE metric using 

NCE = 21.06Z + 50. 

The four correlation pattern runs were based upon a simulated district 
size of 500 cases per grade. Selection was based upon a Z of -.52 corres- 
ponding to the 30th percentile. Therefore, there were approximately 150 
cases in each selected group. The means and standard errors reported in 
this paper were based upon a sampling distribution of 300 weighted averages. 
The range of sample sizes ran from 120 to 180. 

Table I shows the population and reproduced sample correlations for each 

of the four runs. These correlations were selected in order to examine the 

effects of high, moderate, and low patterns. Run 2 was selected to demonstrate 

the case in which r = r . 

^0^2 ^0^1 

Table II shows the obtained and estimated regression effects and corres- 
ponding standard errors, for selection to pretest, pretest to posttest and 
selection to posttest by run. In addition, the table shows the differences 
between obtained and estimated posttest means and corresponding standard 
errors. As would be anticipated, the magnitude of the correlations is 
inversely related to the amount of regression and to the size of the standard 
errors. 

Run 1 represents the lowest set of correlations and consequently the 
greatest anount of regression. The pretest to posttest regression was just 
a little under 5 /s. The regression estimates were all quite close to 
the obtained values. 

ERIC ^ ^ 



TABLE I 

Population and Reproduced Sample Correlations by Run 

Sel-Pre Pre-Post Sel-Post 

Pop. .7 -6 .5 

Run 1 Sample .7017 .5982 .5006 



Pop. .6 .7 .6 

Run 2 Sample .6016 .7017 .5981 



Pop. 



,8 .8 .6 



Run 3 Sample .8001 .8000 .5997 



Pop. .9 .9 .8 

Run h Sample .9003 .9002 .8001 



i 
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TA15LK II 

OBTAINED AND ESTIMATED REGRESSIOH HITH STANDARD ERRORS BY RUN 
(VALUES ARE EXPRESSED IN NCE UNITS) 



Run 1 

Obtained Regression 

Amount S . E . 

Sel-Pre 7.177 1.342 
Pre-Post 4.830 1.575 
Sel-Post 12.006 1.660 



Estima ted Regression 
Amount S.E. 



7.221 
4.969* 
12.146 



1.308 
1.320 
.476 



(Zo - ) 



Ratio 



Amount 



-.140 



S.E. 



1.553 



Mult. Linear 



Amount 



-.085 



S.E. 



1.407 



Run 2 



Obtained Regression 

Amount S.E. 



Sel-Pre 9.611 
Pre-Post .105 
Sel-Post 9.715 



Ratio 



1.306 
1.260 
1.396 



(Z, - 7;2> 



Estimated Regression 
Amount S.E. 



9.728 

.117* 
9.728 



.385 

1.292 
.385 



Amount 



S.E. 



Mul t . Linear 
Amount S.E. 



-.013 



1.339 



.05 



1 .126 



i2 



1 1 

TABLE II (continued) 



Run J 

Obtained Rey.ression 
Amount 



Sel-Prc 4.942 
Prc^-Post 4.867 
Sel-Post 9.790 



Rat io 



Amount 



.037 



1.047 
1.063 
1.475 



S.K . 
1 .426 



U2 - Z2> 



Estimated RcRression 
Amount S . E . 



4.877 
4.812* 
9. 753 



.177 
1.042 
.355 



Mul t . Linear 
Amount S.E. 



-.021 



1 .04 7 



Run 4 



Ob ta inocl Routes s itm 

Amount S.E. 



Sf-l-Pre 2.485 
?re-Post 2.421 
Scl-Post 4 .906 



. 754 
.769 
1 . 086 



Estimated Regression 
Amount S . E . 



2.438 

2.391* 

4.877 



.089 
.752 
.177 



Ratio 



Amount 



.03 



S.E, 



1.069 



Mul t . Linear 



Amoun t 



.015 



S.E. 



764 



♦This value represents the ratio estimate 
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For both the ratio and multiple 1 iticar csnimaLors, there was a sM^;ht 
tendency to overostimatG the no-treatment expectation when comparing obtained 
and expected posttest means. The mngnitude of this overes tima t ion is so 
small that it is of little pragmatic importance. On the other hand, the 
standard errors of both estimators appear to he too large to risk their use 
in estimating no- L rea I men t expectations. That is, it would be quite possihl.- 
for an LF.A with 500 students or less per grade to underestimate Title 1 
treatment effects by \s much as two or three NCI-'s Using either estimator. 

Run 2 also reflects relatively low correlations, however, in this case 

tfie population correlations were selecteti such that r ,^ = r^ . As can 

' ()' 2 i)] 

i)e seen, the amount of statistical regression from pretest to posttest is 
onlv about cuie-tenth of .in NCE. Again the two estimators are quite aciMirate 
in tfuM'r }dreca*it. Although this result does demonstrate that pretest . ^i 
posttt'.t regression is virtu/illv eliminated when r ^ " '^x x regardle>;s of 
the ^izv of r , the standard errors asso( iated with (Z, - '/.^) an* again 
too larg^e t(^ recommend that either (estimator be used. In this case, under- 
t^^Uimates of treatment effect as great as two NTF/s are still quiti- possifile. 

In Run 3 the correlations arc moderately high with the exception of 
r . And since this correlation is critical to the pretest to posttest 
regress ion , it Is not surpr is ing to f ind n pretest * post test regress i on 
of .limost 5 KC[ ' H wevcr, the standard errors are nil slightlv smaller 
due to Ihr inm\i'-* - magnitude of the other cor n* la t ions . 

It is also interesting to note that in this c.ise both estimators prodticed 
sampling -.an cUfferonces between obtained ari<! eslinated pofUtest means of 
less tha»; 4/100 of an NCiK. Although there is a slight dr<^p in the standard 
errors associated with {'7.^ - Z^) in this run, thev are still too large to 
risk using either estimator. 
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Finally, in Kun 4 the correlations are relatively hi^h with ^^^x^ 
at .8. In this case the pretest to posttest regression has dropped to 2.4 
NCE's. The standard errors associated with (Z^ - are also smaller than 

those in runs one through three. Unfortunately, even these standard errors 
could result in underestimating the treatment cifects by as murh as 1.5 to 
2 NCE's depending on the estimator used. 

The results of this limited simulation indicate that, at least for 
districts with 500 or fewer students per grade and correlations of .9 
or lower, the ratio and multiple linear estimators do not provide a viable 
option for obtaining an unbiased estimate of the no-treatment expectation. 
The chances of over estimating the no-treatment expectation are too great 
and consequently too Tiiany LEAs could .nd up seriously underestimating their 
treatment effects. On the other hand, it is possible that, for districts 
with 2,000 or more students per grade, either estimator would produce estimates 
with sufficiently small stnn lard errors. 

Anotlier, and perhaps more viable option, is to use local correlations 
rather than re(;uiring population correlation. In this case, we would be 
'lefining the population mean toward which low scoring students would 
regress as the district mean per grade. This might be a particularly 
attractive* approach when the district means are close to the national 
means. One drawback to this approach would be that correlations would 
have to be based on non-Title I participants. As a result, there would 
be some attenuation dn- to restriction in range. However, the estimates 
are still likely to be more accurate and less variable than when depending 
upon nati(^nal correlation data. 
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